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w  2012  AFOSR  SPRING  REVIEW 


NAME:  Fariba  Fahroo 


BRIEF  DESCRIPTION  OF  PORTFOLIO: 

Discover  Mathematical  Algorithms  and  Develop  Computational 
Techniques  that  Provide  More  Accurate,  Reliable,  and  Efficient 
Algorithms  for  Modeling  and  Simulation  and  Design  of  Complex  Systems 
for  the  U.S.  Air  Force  with  radical  cost  and  turn-time  improvements. 


LIST  SUB-AREAS  IN  PORTFOLIO: 

Multi-Scale  Modeling  ( plasma,  structures,  fluids,  combustion)^* 
Multi-Physics  Modeling  (fluid-structure  interactions,  particle-fluid 
models) 

Uncertainty  Quantification 

Multidisciplinary  Optimization  and  Control  (Design  Under 
Uncertainty) 
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w  Transformational  Opportunities 
*if  for  Computational  Math 


•  The  next  generation  of  computing  systems  based  on  emerging 
heterogeneous  architectures  offer  unprecedented  capabilities  for 
computational  science.  ^  Predictive  High-Fidelity  Simulations 

•  Increase  in  computing  power  isn’t  everything 

•  Challenges : 

>  Constraints  on  system  Power 

>  slower  growth  in  Memory  bandwidth  and  capacity, 

>  Cost  of  data  movement , 

>  Slower  growth  in  clock  rate  and  move  to  Concurrency  of  higher 
number  of  nodes  and  threads, 

>  Slower  growth  in  I/O  bandwidth, 

>  Resiliency  and  Reliability  of  the  computing  systems 

Addressing  the  challenges  in  Computational  Science  for  these  emerging 
computing  systems  at  these  scales  (peta,  exa)  require  fundamental 
paradigm  shift  in  algorithms,  software,  architecture  Hocinn 
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V  ^Challenges  in  Computational  Math 

The  Role  of  Mathematics 


Goal:  Simulation,  Analysis  and  Design  of  complex  systems  with  radical 
cost  and  turn-time  improvements  —  Need  for  Accurate,  Reliable,  and 
Efficient  Algorithms 

•  Multi-Scale  Problems  (spatial  and  temporal)  —  Not  possible  to 
resolve  all  scales,  so  there  is  need  for  Modeling  and  passage  of 
information  across  scales 

•  Multi-Physics  Problems  —  Just  coupling  individual  components  won’t 
do  —  proper  Interface  formulation 

•  High-Dimensional  Problems  —  the  curse  of  dimensionality  won’t  be 
mitigated  fully  by  more  computing  power  ->  more  math  modeling  is 
needed 

•  Effect  of  Uncertainty  —  ubiquitous,  high-dimensional,  if  not  managed, 
detrimental  to  accurate  analysis  and  design 

•  Effect  of  mesh  and  geometry  —  heart  of  discretization  and 
algorithms,  bad  meshes  lead  to  major  inaccuracies,  need  for 
consideration  of  complex  geometry 

•  Scalability  of  Algorithms  —  parallelism  -  New  Challenges  for 
emerging  architectures 
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•  Support  of  High-Order  Methods  of  Accuracy: 

-  An  interplay  of  Efficiency , Accuracy  and  Robustness — 
application  areas  in  Structures,  Plasma,  Combustion 

•  Highlight  of  CFD  Efforts  (2  YIP  Projects,  AIAA  Workshop), 

•  Transition  Stories 

•  Multiscale  Methods:  Structures 

•  Uncertainty  Quantification: 

-  High-Dimensional  Problems 

-  Bayesian  framework  for  UQ  ->  a  MURI  2012  in  Material  Science 

-  Design  Under  Uncertainty  a  2012  BRI  topic 
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V  >  Challenges  for  High-Order 

V  Methods 

•  Low-order  methods  suffer  from  undesirable  excessive  numerical 
dissipation,  and  explicit  time-integration  methods  have  severe  time-step 
restrictions  for  stability  for  multiscale,  multi-physics  simulation  need  for 
high-order  accurate,  implicit  methods.  BUT 

•  High-Order  methods  are  not  as  robust  as  lower-order  ones  -  convergence 
to  steady-state  can  stall  or  be  very  slow. 

•  Time-  Integration  techniques  -  explicit  methods  too  expensive  -  need 
speed-ups  in  implicit  methods  (expensive  Jacobians  in  time-integration  and 
preconditioning) 

•  Robust,  compact,  accuracy  preserving  and  convergent  limiters  (for 
problems  with  strong  discontinuities) 

•  Low  storage,  efficient  iterative  solution  methods  including  hp-multigrid 
methods  for  high-order  operators  YIP  10  effort  exploring  sparsity  in  large 
expensive  Jacobians 

•  High-order  viscous  grid  generation,  error  estimate  and  anisotropic  hp- 
adaptations  YIP  11  effort  using  adjoint-based,  output  error  methods  for 
adaptation 
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^  k  Sparse  Line-Based  High-Order  Methods 

(Discontinuous  Galerkin)  on  Unstructured  Meshes: 
Per-Olof  Persson,  UC  Berkeley  (YIP  2010) 


Goal:  Reduce  the  cost  of  implicit  solvers  by 
using  sparsity  in  large,  expensive  jacobians 

Line-DG:  Apply  1-D  DG  schemes  separately 
along  each  coordinate  line 


•  Magnitudes  sparser  Jacobians 

•  Simple  scheme:  Only  1-D  integrals 


32  81  160  275  432  637  896  12L5  1600  2057  i  is  unlimited.. 


7 


Sparsity-preserving  Quasi-Newton 
Implicit  RK  Time-stepping 


•  Implicit  methods  need  to  solve  systems  (I  -  uAtA)AKi  =  G 
with  second-order  Jacobian  matrices  A  =  Kn  +  K12K2i 

•  Hard-wire  GMRES  matrix-vector  products  to  retain  sparsity 


Use  quasi-Newton 
solvers  to  reduce  the 
number  of  expensive 
Jacobian  assemblies 

Majority  of  time  spent 
in  residual  evaluation 
-  an  implicit  solver  for 
about  the  same  cost 
as  an  explicit 


recomputed 


100 


200 


250 


150 

Quasi-Newton  solve  # 
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Variable-Order  Adaptation  for  Unsteady 

Flows:  Kris  Fidkowski,  Univ  of  Michigan  (YIP  2011) 


Goal:  Improve  robustness  and  efficiency  of  unsteady  simulations  through 
output-based  error  estimation  and  space-time  mesh  adaptation 

Why  Outputs? 

Output  =  scalar  quantity  computed  from  the  CFD  solution 

•  A  CFD  solution  may  contain  millions  of  degrees  of  freedom 

•  Often  of  interest  are  only  a  few  scalars  (forces,  moments,  etc.) 

•  It  is  mathematically  easier  to  speak  of  “error  in  an  output”  than  “error  in  a  CFD 
solution” 

Output  error  =  difference  between  an  output  computed  with  the  discrete 
system  solution  and  that  computed  with  the  exact  solution  to  the  PDE 

Output  error  estimation 

•  Identifies  all  areas  of  the  domain  that  are  important  for  the  accurate  prediction  of 
an  output 

•  Accounts  for  error  propagation  effects 

•  Requires  solution  of  an  adjoint  equation 
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Variable-Order  Adaptation  for 
Unsteady  Flows 


Approach: 

Discontinuous  FEM 

Dynamic  spatial  order 
refinement 

Adjoint-based  error 
estimation 

Promising  Results  but 
solving  adjoint  equations  is 
expensive 


Q' 


time  slab  Xn.k 


dynamic  spatial  order 


element  7w,e 


Time  integral  of  airfoil  lift  coeff 


4th  adaptive  iteration:  blue=1,  red=5 


Airfoil-vortex  encounter 
adaptation: 
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i  i  An  Entropy-Adjoint  Refinement 

Indicator 


Goal:  Analyze  properties  of  a 
cheap  output-based  indicator 
derived  from  an  entropy-adjoint 
connection 
Approach: 

Discontinuous  FEM 
Steady  and  unsteady  problems 
Spatial  and  temporal  refinement 
Key  Ideas: 

Use  entropy  variables  as  unsteady 
adjoint  solution  for  an  entropy 
balance  output 

Adapting  on  this  output  minimizes 
spurious  entropy  generation  and 
produces  “all-around”  good 
solutions 


Unsteady  shallow-water  equations 


example 


Initial  condition 


Solution  at  t  =  tf/2 


Output  adjoint  at 
t  =  tt/2 


/i-refinement 


Output  adjoint  Entropy  adjoint 
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1st  International  Workshop 
on  High-Order  CFD  Methods 


•  Workshop  held  at  the  2012  AIAA  ASM  meeting: 
a  tremendous  success 

-  30  groups  from  all  over  the  world  made  over  70 
presentations 

-  80  global  participants  from  ~12  countries  -  9  Pis 

-  Support  for  students  provided  by  AFOSR 


Workshop  Summary 


*  14  different  test  cases  were  considered. 

*  For  smooth  inviscid  and  viscous  flow  problems,  high- 
order  methods  outperform  2nd  order  production  codes 
in  terms  of  error/CPU  time 

*  Solution  based  grid  and  accuracy  adaptations  highly 
effective  in  achieving  solution  efficiency  and  accuracy 

*  Further  progresses  needed  for  high-order  methods  to 
efficiently  and  robustly  solve  the  Reynolds  averaged 
Navier-stokes  equations  with  turbulence  models 

*  Research  on  high-order  mesh  generation  with  curved 
boundary  is  needed 

*  The  2nd  Workshop  will  be  held  in  Cologne  in  the 
summer  of  2013 
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Recent  Transitions 


>  PI:  (Suresh  Menon  ,  GATech) 
Hybrid  Algorithms  for  LES  of 
Multi-Phase  Flows  -Transition 
to  RW 


>  PI:  (Marsha  Berger,  NYU) 

Inclusion  of  the  Adaptation/Adjoint 
module,  Embedded  Boundary 
Methods  in  the  software  package 
Cart3D  —  Transition  to  NASA,  ONR, 
DOE,  AFRL,  DIA 


Application  to  Explosively 
Formed  Projectile  (EFP) 


Cart3D  used  for  computing 
Formation  Flight  to  reduce  drag 


and  improve  energy  efficiency 
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A  Generalized  Finite  Element  Method  for 
Multiscale  Simulations  -  C.A.  Duarte,  UIUC 


•  Objectives:  Develop  and  analyze  a  computational  method  able  to 
capture  multiscale  phenomena  in  structural-scale  FEM  meshes 

•  Approach:  Generalized  FEM  with  global-local  enrichment  functions: 
GFEM9' 


Goal:  Capture  fine  scale  effects  on  coarse  meshes  at  the  global  (structural) 
scale 
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^  t  A  Generalized  Finite  Element  Method  for 

MultiSCale  Simulations:  Armando  Duarte  (UIUC) 


Load 


Extensions  to  Nonlinear 
Problems: 


Load 


Key  Properties: 


Local  Problem 


„ 

Enriched  Global 


Initial  Global 
Problem 


Problem 


•  Uses  available  information  at  a  simulation  step  to  build  approximation  spaces  for  the  next  step 

•  Uses  coarse  FEM  meshes;  solution  spaces  of  much  reduced  dimension  than  in  the  FEM 

•  Two-way  information  transfer  between  scales;  account  for  interactions  among  scales 
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A  Generalized  Finite  Element  Method  for 

Multiscale  Simulations 


Example:  Simulation  of  3-D  Propagating  Cracks 


(a)  step  0  (b)  step  20  (c)  step  35  (d)  step  50  (e)  step  70 


(f)  step  85  (g)  step  93  (h)  step  103 


(i)  step  1 10 


(j)  step  146 


FAST:  Coarse-scale  model  of  much  reduced  dimension  than  FEM;  Fine-Scale 
computations  are  intrinsically  parallelizable;  recycle  coarse  scale  solution 


ACCURATE:  Can  deliver  same  accuracy  as  adaptive  mesh  refinement  (AMR)  on 
meshes  with  elements  that  are  orders  of  magnitude  larger  than  in  the  FEM 


■  STABLE:  Uses  single-field  variational  principles 

■  TRANSITION:  Fully  compatible  with  FEM 
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Uncertainty  Quantification  MURI 
Broad  MURI  Objectives 


>To  develop  rigorous  theory,  algorithms  and  software  for  UQ  management  in  a 
computationally  scalable  way. 

>  Challenges  include 

>proper  formulation  of  stochastic  models 
>high-dimensional  problems 
> Proper  numerics:  efficiency,  accuracy 

Challenging  application  problems:  stochastic  multiscale  problems 
>  Need  for  new  frameworks  for  dealing  with  model  uncertainty,  combining 
epistemic  and  aleatory  uncertainties 
> Inverse  Problems 
>Design  and  Control  problems 


Research  Highlights 


>  Mathematical  Theory:  Quantization-renormalization  of  SPDEs;  New  evolution 
equations  for  joint-pdf  of  SPDEs;  Nonlinear  Malliavin  calculus 

>Reduced  Basis  Methods  (RBM):  Integral  equations  and  multi-scattering  problems; 
Robust  design,  parameter  estimation,  and  model  uncertainty 

>  Adaptive  ANOVA:  Convergence  theory;  Parameter  compression  and  RBM;  Fluid 
flows,  porous  media,  multi-scattering 

>  Bayesian  Framework:  coarse-graining;  Active  learning  +  SPDEs;  Adaptive  SMC, 
dependent  random  variables,  model  uncertainty  in  inverse  problems 

>Numerical  SPDEs:  Data-driven  stochastic  multiscale  method;  Multiscale,  multilevel 
MC,  long-time  integrators  of  SPDEs 

>Software:  MEPCM  library;  Reduced  basis  method  libraries  -  RBOOMIT,  RBAppMIT; 
Random  poly-crystals  -  RPCrystal 
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New  stochastic  modeling  methods  are  required  for  time- 
dependent  and  high-dimensional  problems 


Time 


0(  number  of  evaluations  for  1e-012  accuracy,  r  =  5) 


•  Uncertainty  increases  in  time 


Curse  of  dimensionality 


New  Developments:  Adaptive  ANOVA,  Multi-Level  MC,  New  joint  PDF  Theory, 

Time-Dependent  Chaos  Basis,  Quantization/Normalization 
Main  Result:  New  methods  beat  Stochastic  Collocation  (standard)  by 

several  orders  of  magnitude. 
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Adaptive  ANOVA  for  100-Dimensional 

Problems 


□  Hierarchical  decomposition  of  a  multidimensional  function  into  combinations  of 
functions  of  subgroups  of  its  dimensions  (allow  only //-body  interactions) 


,  X]S[ ) 


N 


N 


^  ^  (Xji  )  +  y  '  ^pfji  J2  ( X3l  ?  XJ2  )  H" 

jlt(i/=l)  3l<32,(y=2) 

N 

^  ^  , 32,32  {Xjl :  Xj‘2 :  X32  )  Jn  iX3l  7  *  *  *  7  ^  jiv  ) 

ji  '32  'js  (  )  order  of  Polynomial  Chaos  approximation 


//  :  dimension  of  each  sub-problem 
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Adaptive  ANOVA:  100-dimensional 

Flows 

♦  >Ref:  Yang,  Choi  &  Karniadakis,  JCP,  2011 


Thermal  convection  in  a  closed  cavity  with  random 
boundary  conditions 


V[u] :  variance 


MC  (90,000 
samples) 


Sparse  Grid 
Level  2  (#:18625) 


ANOVA 

[i=2,  v=l  (#:193) 
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Bayesian  Framework  for  UQ: 

Zabaras(Cornell) 


Builds  surrogate 
surface 


Informative  data  collection 


Problem  set  up: 

1 .  Deterministic 
(multiscale)  solver 

2.  Limited  data. 

3.  Stochastic 
discontinuities. 

4.  Multiple  outputs 


i 


Bayesian  Methods 
(GP,  RVM,  etc.) 


Error  bars  for  the 
statistics 


RVM,  N=1 0,5=1 0_1 
RVM,  N=1 0,6=1 0-2 
RVM,  N=1 0,5=1  O'3 
RVM,  N=1 0,6=1  O'4 
- MC 

. i . .iii, . 

1  . 

. Llftj 

\  \,li 

kjij 

. :iy .. 

M  T 

"w 

hi 

t  •: 

At  j 
/.  *  ! 

/■;  j 

Jy 

Hierarchical  binary 
trees 


Capturing 

correlations 


(a)  K=8,  MGP  (N  =  20 ,6  ■ 
std.  of  u 


10"3): 


(b)  K=8,  MC:  std.  oft 


(a)  Elements:  6  =  10“4 


(b)  Observed  input:  S  =  10“4 
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Bayesian  Exploration  Statistical 

Toolbox  C++ 

“Designing  the  Next  Generation  Exascale  Statistical 
and  Uncertainty  Quantification  Tools” 

□  Objectives: 

>  High-dimensional,  data-driven  exploration  of  PDE-based  engineering  and 
science  problems 

>  Fast  implementation  and  testing  of  novel  statistical  algorithms 

□  Key  ideas: 

>  Modularity: 

boost 

C  +  +  LIBRARIES 

>  Efficiency/Scalability: 

BLAS  ScaLAPACK 

L  A  P  A  C  K 
L  -A  P  -A  C  -K 

L  A  P  A  -C  -K  ^ 

L  A  PAC  K  OPT++ 

L  A  -P  -A  C  K 
L  -A  -P  A  C  -K 


□  Features: 

>  Optimized  templated  versions  of 
core  statistical  models 

>  Data-based  input  modeling 

>  Uncertainty  propagation 

>  Inverse  problems 


Free  as  in  Freedo 
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Adaptive  Methods  for  Stochastic  PDE 


Constrained  Optimization:  M.  Heinkenschloss  (Rice 


Optimal  control  and  design  problems 
uncertain  inputs. 

Proper  incorporation  of  uncertainties 
decisions. 

Resulting  optimization  problems 
require  ‘sampling’  of  uncertainties 
and  are  expensive  to  solve. 

Expense  increases  with  sample  size. 

Developed  optimization  approach 
based  on  adaptive  sparse  grids: 

-  Importance  of  the  random  inputs  on 
optimization  detected  automatically. 

-  Sampling  is  adjusted  to  keep  number  of  PDE 


solves  at  minimum  while  ensuring  convergence. 


•• 
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•  :•  • 

k  •  J  •  *  •  •  t  *• 

*  •  i  • 
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Adaptive  sampling  (right)  substantially 
reduces  sample  size  over  classical  sparse 
grid  samples  (left) 


Expected  Value  of  State 


Standard  Deviation  of  State 


-  Implementation  using  Trilinos  for  parallelism.  Source  inversion  problem.  Expected  value 
**  m  _ .  ....  *  .  (left)  and  standard  deviation  (right)  of  PDE 

Adaptive  sampling  leads  to  reduction  S0|Uti0n.  Large  STdev  near  left  bdryjue  to 
in  computing  time  by  a  factor  of  15  to  20.  uncertain  inflow. 
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Future  Directions 


•  Continued  support  of  high-order  methods  in  multiscale  and 
multiphysics  modeling,  support  in  time-integration  methods 

•  Continued  emphasis  on  UQ  and  V&V 

-  Numerical  methods,  Sampling  methods,  Design  under 
Uncertainty 

•  Algorithms  for  multi-core  platforms:  GPU  computing 

•  Emphasis  on  scalability  of  algorithms  for  ultra-parallel  large  scale 
computing 

•  Data-Driven  Modeling  and  Computation 

-  Combination  of  new  estimation  techniques  with  large  scale 
computing:  new  ideas  in  data  assimilation 

•  Understanding  the  impact  of  geometric  discretization  on  the 
fidelity  of  the  analysis  and  computation 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited.. 


26 


